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FOREWORD 


Currently  the  Standard  Automated  Materiel  Management  System  (SAMMS)  uses  a 
single  technique  to  forecast  demand  for  all  hardware  items.  To  evaluate, 
through  implementation,  the  potential  for  improved  forecasts  using  a  variety 
of  demand  forecasting  techniques,  a  Requirements  Forecast  Working  Group  (RFWC) 
was  created  by  the  Supply  Management  Division.  The  Defense  Logistics  Agency 
Operations  Research  and  Economic  Analysis  Management  Support  Office  (DORO),  as 
a  member  of  the  RFWG ,  was  tasked  with  an  initial  study  of  the  methodology 
governing  the  use  of  a  variety  of  forecasting  techniques  and  the.  potential 
benefits  of  improved  demand  forecasts  for  DLA  managed  items. 

The  authors  of  this  paper  are  indebted  to  RFWG  participants  who,  through  their 
comments,  questions  and  ideas,  have  kept  us  on  the  right  path  which  enabled 
the  timely  completion  of  this  study. 

We  are.  also  indebted  to  the  Defense  General  Supply  Center's  Office  of  Planning 
Resource  Management  for  their  expertise  in  formulating  the  initial  concepts 
into  a  study  that  could  be  successfully  completed  and  implemented  within 
current  DLA  policy. 

L 

CHRISTINE  L.  GALLO 
Deputy  Assistant  Director 
Policy  and  Plans 
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EXECUTIVE  SUMMARY 


The  Defense  Logistics  Agency  ( DLA)  currently  employs  the  Standard 
Automated  Materiel  Management  System  (SAMMS)  under  a  mode  which  forecasts 
demand  for  all  Quarterly  Forecast  Demand  (QFD)  Items  through  the  use  of  a 
single  forecasting  technique.  This  approach  for  these  QFD  items  has  been 
shown  through  previous  analysis  to  result  in  long  term  forecasting  errors. 
The  result  of  these  errors  in  forecasts  is  that  DLA  has  consistently 
maintained  higher  safety  levels  which  has  contributed  to  the  presence  of 
excess  on-hand  stocks. 

Consequently,  the  overall  thrust  of  this  analysis  has  been  to  enhance  the 
forecasting  methodology  of  SAMMS  by  exploring  alternative  forecasting 
techniques  which  would  have  the  potential  to  enhance  the  accuracy  of  long 
term  forecasts.  These  improvements  could  be  accomplished  through  the 
development  of  an  approach  which  would  reduce  the  lead  time  forecasting 
errors  inherent  to  the  system.  As  important  as  these  goals  are  today,  they 
will  be  of  greater  concern  in  the  near  future  as  DLA's  mission  expands 
through  the  projected  transfer  of  1.4  million  items  currently  maintained  by 
the  TRI -Services  and  is  reflected  by  DLA's  Strategic  Plan  under  Materiel 
Management  (i.e.,  Objective  3)  which  is  to  sustain  customers  in  the  most 
effective  and  efficient  manner. 

The  project  has  succeeded  in  the  development  of  a  multiple  forecasting 
methodology  which  has  the  capability  to  select  the  most  appropriate 
forecasting  technique  for  each  QFD  item.  The  study  has  found  that  a  system 
improvement  of  9  percent  over  the  current  SAMMS  method  could  be  achieved 
in  forecasting  accuracy.  It  was  estimated  that,  while  maintaining  a 
constant  supply  availability  rate,  a  $42  million  non-recurring  cost  savings 
could  be  realized  by  implementation  of  this  technique  as  an  adjunct  to 
SAMMS.  These  projected  savings  could  substantially  increase  once  the 
consumable  item  transfer  of  1.4  million  items  for  the  TRI -Services  is 
accomplished  with  implementation  of  this  methodology  supporting  the  Defense 
Management  Review  Decision  (DMRD)  #901  in  the  goal  of  reducing  supply 
costs. 

Specific  recommendations  stemming  from  this  project  are  the  following: 

►  Enhance  the  SAMMS  forecasting  methodology  with  a  multiple 
forecasting  model  for  QFD  items  as  an  adjunct  to  SAMMS. 

►  Implement  the  Multiple  Forecasting  Model  at  the  Hardware  Centers 
and  for  Medical.  Use  a  staged  process  under  the  guidance  of  the 
Requirements  Forecast  Working  Group  (RFWG)  chaired  by  the  Supply 
Management  Division.  These  stages  will  entail  the  following: 

►►  Develop  a  SAMMS  linked  prototype  model. 

►►  Test  the  prototype  model  at  a  selected  Hardware  Center. 

►►  Extend  the  prototype  model  to  all  Hardware  Centers  and  to 
Medical  under  full  implementation. 


i.  mrRMwenaN 

A.  Background.  Currently  the  Standard  Automated  Materiel  Management 
System  (SAMMS)  uses  a  single  technique  to  forecast  demand  for  all  hardware 
items.  SAMMS  forecasts  are  being  adjusted  by  supply  center  with  locally 
developed  forecasting  programs  and  manual  inventory  manager  intervention. 
Previous  DLA  studies  have  focused  on  finding  a  single  technique  to  improve 
forecasting  for  all  items.  A  demand  forecasting  study  of  subsistence  items 
has  found  that  a  set  of  four  simple  methods  could  perform  better  than  the 
current  subsistence  techniques. 

B.  Purpose.  The  purpose  of  this  study  was  to  define  the  criteria 
for  a  multiple  forecasting  model  to  operate  in  conjunction  with  the  current 
SAMMS  system  and  data  base.  The  multiple  forecasting  model  optimally 
selects  the  most  appropriate  forecasting  technique  for  each  item. 

C.  Objectives.  The  objectives  of  this  study  are: 

1.  Identify  appropriate  forecasting  techniques  and  evaluation 
methodology. 

2.  Define  the  data  requirements  for  obtaining  the  best  forecast 
for  each  item. 


3.  Provide  recommendations  for  implementation  strategies. 

D.  Scope  of  study. 

1.  Demand  based  replenishment  items  that  are  established  and  are 
currently  forecasted  on  a  quarterly  basis  were  used. 

2.  Subsistence,  fuels,  clothing  and  textile  items  were  not 

included. 

3.  Forecastirg  techniques  studied  were  limited  by  their  ability 
to  forecast  with  a  maximum  of  20  quarterly  data  points  allowing  a  minimum 
of  the  latest  4  quarters  for  forecast  evaluation. 

4.  Items  included  in  this  study  had  24  quarters  of  demand 
history,  where  zero  demand  is  an  acceptable  history. 

II .  LITERATURE  OVERVIEW 

Research  on  potential  forecasting  techniques  for  use  in  a  multiple 
forecasting  system  has  focused  on  the  ability  to  initialize  and  forecast 
with  as  little  as  12  quarters  of  demand  data. 

It  was  not  the  intent  of  this  study  to  evaluate  each  forecasting  technique 
on  the  basis  of  previous  empirical  testing  since  testing  can  be  conducted 
to  favor  one  technique  over  others.  A  key  assumption  is  that  some  methods 
are  better,  depending  upon  the  characteristics  of  the  time  series  used  for 
forecast  evaluation  (2  p66) . 
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Several  key  concepts  were  kept  in  the  forefront  of  this  project  as  research 
was  conducted  and  various  statistical  measures  and  formulas  were  evaluated. 
First,  simple  techniques  are  at  least  as  accurate  as  complex  forecasting 
methods  (15  pl54,l6  pi42) .  Second,  even  the  best  statistical  method 
explains  only  a  small  portion  of  the  information  contained  within  the 
demand  stream  (2  p61) . 

Two  areas  of  consideration  with  demand  data  preparation  are  to  check  for 
outliers  and  deseasonalize  the  demand  stream.  Although  there  are  varying 
methods  to  identify  and  adjust  for  outliers  (11),  the  simplest,  and 
possibly  most  effective  method  given  large  variances  in  demand,  is  to  put 
upper  and  lower  limits  on  acceptable  demands  (3) .  For  example  if  a 
quarters  demand  is  outside  the  acceptable  bounds,  set  that  period's  demand 
equal  to  the  limit.  This  approach  will  be  referred  to  as  demand  dampening 
and  is  discussed  in  greater  detail  in  section  IX. C. 

Deseasonalizing  the  data  has  not  been  shewn  to  increase  forecast  accuracy 
(20,1  pl73) .  Also  since  this  study  includes  techniques  designed  to 
forecast  demand  data  with  seasonal  characteristics,  this  type  of  data 
preparation  was  not  necessary. 

The  combination  of  competing  forecasting  techniques  is  receiving  a  great 
amount  of  attention  in  current  literature.  Most  agree  that  a  better 
forecast  can  be  obtained  by  combining  the  individual  forecasts  of  two  or 
more  techniques.  This  may  be  more  appropriate  when  the  forecasting  horizon 
is  longer  (3  pl84) .  To  combine  forecasts,  methods  for  computing  the  weight 
of  each  forecast  are  varied.  An  appropriate  technique,  which  again  is  the 
simplest,  is  to  use  the  arithmetic  mean  of  the  individual  forecasts  (2 
p58, 15  pl55, 16  pl39) .  This  was  the  approach  undertaken  for  this  study. 

III.  MULTIPLE  FORECASTING 

A.  Assumptions.  The  concept  of  a  multiple  forecasting  model  centers 
on  the  following  two  assumptions  which  are  based  on  work  introduced  by 
Smith  (24) . 

1.  One  forecasting  technique  will  be  the  best  for  a  given  item. 

2.  The  technique  that  worked  best  in  the  past  will  work  best  in 
the  future. 

B.  Description.  A  forecasting  model  was  designed  from  these  rather 
simple  ideas.  Consider  an  item  with  20  quarters  of  demand  history.  The 
demand  stream  is  broken  into  three  parts.  The  first  12  quarters  are 
defined  as  the  initialization  period.  The  initialization  period  is  used  to 
establish  any  constants  or  parameters  associated  with  a  forecasting 
technique.  Regression  and  smoothing  techniques  use  all  demand  points  in 
the  initialization  period.  Naive  techniques,  such  as  the  last  demand,  use 
only  one  demand  point. 
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The  second  part  is  the  testing  period.  This  represents  the  demand  that  is 
forecast  using  each  of  the  available  forecasting  techniques.  Forecasting 
error  and  bias  are  measured  for  each  technique.  Finally,  from  the  criteria 
established  by  this  study,  one  forecasting  technique  is  selected  to 
forecast  an  item's  future  demand. 

The  last  4  quarters  are  the  results  period.  Only  the  selected  technique 
was  used  to  forecast  demand  for  this  time  period.  Forecasting  accuracy  of 
the  model ' s  selected  technique  was  compared  to  the  accuracy  of  a  SAMMS 
forecast.  The  length  of  the  results  period,  which  was  defined  through 
discussion  within  the  RFVJG,  is  1  year.  The  basis  for  this  decision  is  in 
the  length  of  item  lead  times  and  procurement  cycles.  One  year  roughly 
represents  the  average  lead  time  plus  1  quarter  (see  Table  1  in  Section 
VII) . 

To  obtain  an  annual  forecast  for  the  results  period,  the  QFD  from  the 
selected  technique  was  modified  in  the  fol lowing  ways: 

1.  QFD  *  4.  This  method  is  similar  to  the  SAMMS'  lead  time 
demand  computation. 

2.  QFD  *  4  +  short  term  trend.  The  short  term  trend  is  computed 
by  the  average  testing  period  error. 

3.  QFD  +  long  term  trend.  Iha  long  term  trend  is  based  on  the 
slope  of  a  regression  line  through  the  initialization  and  testing  period 
demands. 


4.  Forecast  from  a  forecast.  This  involves  using  the  QFD  as  a 
bridge,  or  the  next  demand  point,  to  obtain  the  next  QFD.  Four  QFDs 
computed  in  this  manner  would  then  be  used  as  the  annual  forecast. 

In  summary,  the  forecasting  model  reads  the  initialization  and  testing 
period  demands.  Any  alteration  of  the  actual  demand,  such  as  percent 
applicable  nonrecurring  demand,  occurs  at  this  point.  Each  of  the 
available  forecasting  techniques  are  used  to  forecast  demand  over  the 
testing  period.  Forecasting  error  and  bias  are  measured.  The  criteria 
established  by  this  study  will  then  be  used  to  select  one  forecasting 
technique  to  forecast  future  item  demand.  Forecasting  accuracy  of  the 
selected  technique  will  be  compared  to  a  SAMMS  forecast. 

IV.  TESTING  PERIOD 

As  previously  mentioned,  many  events  take  place  during  the  testing  period. 
Forecasts  are  made,  error  and  bias  are  measured,  and  one  technique  is 
selected.  In  this  section  various  methods  for  measuring  forecast  error, 
applying  the  forecast  to  the  testing  period,  and  evaluating  forecast  bias 
are  explored. 
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A.  Forecast  Accuracy  Measures.  Three  testing  period  error  measures 
were  used  to  assess  the  accuracy  of  each  forecasting  technique.  The  Mean 
Absolute  Deviation  (MAD)  is  an  average  of  absolute  forecasting  errors  and 
the  Mean  Squared  Error  (MSE)  is  the  average  of  squared  forecasting  errors. 
The  MSE,  by  squaring  the  error,  unproportionally  places  more  emphasis  on 
larger  errors.  This  means  that  techniques  yielding  smaller  MSEs  decrease 
the  potential  for  large  forecasting  errors .  Total  Absolute  Error  (TAE)  is 
the  difference  between  the  sum  of  the  forecasts  and  the  sum  of  the  demands. 
The  TAE  has  an  expected  value  of  zero  and  could  be  interpreted  as  how  well 
the  future  average  demand  is  represented  by  the  average  forecast  of  demand. 
Error  measures  are  defined  as  follows: 


n 

E  ABSCAt  -  Ft) 
t=l 

MAD  = - 

n 


n 

2  (Aj.  -  Ft)2 
t=l 

MSE  =  - 

n 


n 

TAE  =  ABS  2  (At  -  Ft) 
t=l 


where:  Ft  =  forecast  for  time  t 

At  =  actual  demand  for  time  t 
n  =  number  of  time  periods 

B.  Applying  the  Forecast.  Besides  error  measures,  the  application 
of  the  forecast  to  the  testing  period  takes  three  forms.  A  forecast  that 
is  straight-lined  has  all  subsequent  values  over  the  forecasting  horizon 
equal  to  the  current  QFD.  A  forecast  of  this  type  potentially 
discriminates  against  seasonal  forecasting  techniques.  One-step  ahead 
forecasts  are  computed  for  only  one  period  into  the  future.  A  combination 
of  the  these  two  types  of  forecast  is  called  a  long-term  forecast.  This  is 
actually  a  series  of  long-term  forecasts  made  with  a  leading  time  frame  of 
one  period. 
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C.  Bias  Measures.  Detection  of  a  forecast's  bias  incorporates  the 
use  of  Theill's  bias  proportion.  Using  this  statistic,  which  has  an 
expected  value  of  zero,  a  threshold  limit  must  be  established  to  eliminate 
potentially  biased  techniques.  The  proportion  is  defined  as  follows: 


U  = 


r  A  -  F  i 


L  RMSE  J 
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where:  F  =  average  forecast 

A  =  average  actual  demand 
RMSE  =  square  root  of  the  MSE 


D.  Technique  Selection.  At  least  three  decisions  must  be  made 
before  a  forecasting  technique  can  be  selected  as  the  "best"  technique  for 
a  given  item's  demand.  First,  should  biased  techniques  be  eliminated  from 
consideration?  If  so,  what  detection  threshold  will  be  used  to  establish  a 
biased  technique?  Secondly,  which  of  the  identified  accuracy  measures  will 
be  minimized?  And  finally,  which  method  for  applying  the  forecast  to  the 
testing  period  will  be  used?  The  answers  to  these  fundamental  questions 
are  developed  in  section  IX. A.  However,  the  basic  premise  behind  the 
technique  selection  process  is  as  follows: 

1.  Techniques  that  yield  a  bias  value  beyond  the  established 
threshold  are  eliminated. 

2.  The  technique  producing  the  least  error  is  selected. 

V.  FORECASTING  TECHNIQUES 

The  techniques  studied  for  inclusion  in  the  multiple  forecasting  model  fall 
into  6  classes.  All  are  time-series  forecasting  models  which  means  that 
the  forecast  is  based  on  historical  demand.  No  assunptions  are  made  with 
regards  to  the  applicability  of  any  technique  to  an  individual  demand 
stream.  This  is  particularly  'mportant  for  the  regression  models  and  the 
standard  assumptions  that  traditionally  accompany  them.  A  technique's 
forecast  is  viewed  as  simply  a  prediction,  accurate  or  not,  and  the  bias 
and  accuracy  of  that  prediction  are  subsequently  evaluated. 

The  following  definitions  apply  throughout  the  technique  discussion: 

F^.  --  Forecast  for  time  period  t 

Xf.  -  Actual  demand  for  time  period  t 

A.  Native  methods.  The  following  naive  forecasts  are  easily  computed 
however  these  methods  wo*ild  overreact  to  one-time  surges  or  outliers  in  the 
demand  stream. 
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1.  The  basic  naive  forecast  assumes  that  the  next  period’s 
demand  is  the  same  as  this  period's.  The  forecast  is  generated  as  follows: 


Fttl  “  \ 


2.  The  seasonal  naive  forecast  assumes  that  the  next  period's 
demand  is  the  same  as  the  demand  1  year  ago.  The  forecast  for  quarterly 
demand  with  a  season  length  of  4  quarters  is  generated  as  follows: 

Ft+1  ""  *t-3 

B.  Moving  averages. 

1.  A  simple  moving  average  forecast  is  an  average  of  the  most 
recent  demands.  The  number  of  periods  to  include  in  this  average  is 
generally  a  multiple  of  the  season  length  and  therefore  is  free  of  seasonal 
effects.  The  4  and  8  quarter  moving  average  forecasts  are  respetd . ivoly 
given  by: 


tt 


t. 

X: 

t.-3  1 


/4 


rt+l 


t 

=  S  X; 
t-7  1 


/8 


2.  A  lagged  2  quarter  moving  average  that  is  based  on  auto¬ 
correlation  analysis  was  also  used.  This  analysis  determines  if  there  is  a 
relationship  between  quarters  at  two,  three,  or  four  intervals  apart. 
Autocorrelations  are  computed  for  the  first  four  lags  and  a  95  percent 
confidence  interval  is  used  to  establish  significance.  This  method  will 
employ  the  following  decision  rules: 


then: 


then: 


then: 


(a)  If  the  second  autocorrelation  is  positively  significant 
Ft+1  =  (*t-l  +  Xt-3>/2 

(b)  If  the  third  autocorrelation  is  positively  significant 
Ft+1  =  (^t-2  + 

(c)  If  the  fourth  autocorrelation  is  positively  significant 

Ft*l  =  (Xt-3  +  \-l)/2 

(d)  If  none  of  the  above  ar?  true  then: 

Ftn  -  (*t  +  \-i)/2 
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C.  Exponential  smoothing.  There  are  many  exponential  smoothing 
models  documented.  Each  method  may  attempt  to  compensate  for  changes  in 
the  demand  stream  such  as  trend,  seasonality,  or  level.  With  possibly  the 
exception  of  Winter's  method,  the  class  of  exponential  smoothing  models  is 
noted  for  minimal  data  requirements  and  low  cost  for  computer  time  thus 
making  them  popular  for  large  inventory  systems.  Initial  smoothing  values 
for  these  models  were  generated  by  using  an  average  of  the  first  4  quarters 
of  the  time  series  and  by  using  the  y- intercept  value  of  a  regression  line 
through  the  time  series.  In  addition,  for  each  of  these  techniques  various 
smoothing  parameters  were  tested.  Smoothing  parameters  were  generally 
chosen  between  0.05  and  0.3.  The  higher  the  parameter  the  more  weight  that 
is  given  to  the  current  observation  or  value. 

1.  Single  exponential  smoothing  is  a  weighted  average  of  the 
previous  forecast  and  the  most  recent  demand.  The  forecasting  model  takes 
the  form: 


Ft+1  =  <*Xt  +  (1-  a)  Ft 


2.  Brown's  double  exponential  smoothing  uses  a  smoothing 
equation  to  smooth  the  current  demand  and  a  second  to  smooth  the  smoothed 
value  of  the  first  equation.  This  technique  provides  a  trend  estimate; 
however,  it  is  not  currently  used  in  SAMMS.  The  SAMMS  modification  is  to 
drop  the  last  term,  "b"  from  the  forecast  equation  below.  A  forecast  is 
generated  from  the  following: 


S't  -  oXf  +  d  -  oOS't.-L 
s"t  =  oS't  +  (1  -  a)S"t_1 
at  =  2S't  -  S"t 
bt  =  [a/( 1  -  o)](S't  -  S»t) 

Ft+m  =  ®t  +  ^t10 

where:  S'  =  single  smoothed  quantity 
S"  =  double  smoothed  quantity 
a  =  smoothing  parameter 
m  =  length  of  forecasting  horizon 
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3.  Holt's  double  exponential  smoothing  is  similar  in  principle 
to  Brown's  except  that  it  smooths  the  trend  values  separately  and  with  an 
additional  smoothing  parameter.  The  forecast  is  given  by  the  following  set 
of  equations: 


St  =  aXt  +  (1  -  a)  (St.!  +  b^) 
bt  =  6(St  -  St_1)+  (1  -  S) bt.-L 


Ft+m  =  ^  +  btm 

where:  S  =  smoothed  quantity 
b  =  trend  component 
a  =  smoothing  parameter 
6  =  smoothing  parameter  for  trend 
m  =  length  of  forecasting  horizon 

4.  Winter's  triple  exponential  smoothing  is  the  only  smoothing 
model  to  be  studied  that  directly  deals  with  seasonal  data.  The  method  is 
composed  of  three  smoothing  equations  for  the  level,  trend,  and  seasonality 
of  the  time  series. 

^  =  oXt/It_L  +  (1  -  a) (St.t  +  bt_!) 

bt  =  5 (St  -  St_!)+  (1  -  6) bt_! 

It  =  /BCt/St;  +  (I"/?)  It-L 

Ft+m  =  (^t  +  btra)  It-Lrt-m 

where:  S  =  smoothed  quantity 
b  =  trend  component 
I  =  seasonal  component 
a  =  smoothing  parameter 
6  =  smoothing  parameter  for  trend 
(3  =  smoothing  parameter  for  seasonality 
m  =  length  of  forecasting  horizon 
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5.  Trigg-Leach  adaptive  lesponse  rate  exponential  smoothing 
models  were  designed  to  automatically  adjust  the  smoothing  parameter  used 
in  single  exponential  smoothing.  Parameter  adjustment  is  based  on  forecast 
error  thus  accommodating  a  sudden  shift  in  the  level  of  the  series.  In 
attempts  to  keep  the  smoothing  parameter  from  overreacting  to  occasional 
demand  spikes  or  outliers,  alterations  of  the  original  Trigg-Leach  model 
were  made.  The  two  modified  methods  studied  use  the  following  for 
parameter  estimation: 

et  =  *t  -  Ft 
%  =  /Bet  +  a-flJEt-i 
Mt  53  /3letl+  (l-0)Mt-i 

at  -  |Et  /  Mtl 


wheie:  e  =  forecasting  error 

E  =  smoothed  forecasting  error 
/3  =  smoothing  parameter  for  error 
M  =  smoothed  absolute  error 
a  =  computed  smoothing  parameter 

(a)  Delayed  a  Trigg-Leach  model  uses  the  a  computed  from  a 
previous  forecast  for  the  current  forecast. 


Ft+1  ~  “t-A  +  (1  “  “t-i)Ft 


(b)  Smoothed  a  Trigg-Leach  model  smooths  the  computed  a  to 
dampen  it's  response (19  p7) . 


<*'t  =  ^(“t)  +  (!  ~ 

Ft+1  =  “'tXt  +  (1  ~  a't)Ft 
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D.  Regression  models. 

1.  The  simple  linear  regression  model  applies  ordinary  least 
squares  to  fit  a  straight  line  through  the  available  demand  data.  The 
assumption  is  that  demand  will  remain  relatively  stable  or  increase  or 
decrease  in  a  linear  fashion.  The  forecast  is  generated  as  follows: 

nEXY  -  (EX) (EY) 

b  - - 

nEX2  -  (EX)2 

a  =  (EY)/n  -  (bEX)/n 


Fm  =  a  +  tm 

where:  Y  =  the  period  of  the  time  series 

n  =  the  number  of  quarters  included 
m  =  length  of  forecasting  horizon 

2.  The  nonlinear  regression  model  uses  ordinary  least  squares  to 
fit  a  straight  line  through  the  available  log  transformed  demand  data.  In 
a  strict  definition  this  technique  is  linear  in  its  parameters  and  is 
therefore  a  linear  model.  However  to  differentiate  from  the  simple  linear 
regression  technique  defined  above,  the  word  "nonlinear"  is  used.  This 
technique  applies  an  exponential  curve  to  describe  the  time  series.  The 
forecast  is  generated  as  above  by  first  taking  the  natural  log  of  the 
demand  data.  After  applying  the  computations  the  forecast  is  simply  the 
antilog  of  the  computed  value. 

E.  Probability  models.  To  accommodate  a  large  number  of  zeros  in  a 
time  series,  a  conditional  probability  method  was  studied.  Each  quarterly 
data  point  was  transformed  to  "1"  if  the  demand  was  greater  than  zero  or  a 
"0"  if  the  demand  equals  zero.  The  basis  for  its  use  depends  on 
"clustering"  or  in  other  words,  a  time  series  that  has  a  greater  tendency 
for  a  "0"  demand  following  a  "0"  demand  (similarly  a  "1"  following  a 
"1") (8) .  This  method  incorporates  the  fol lowing  decision  rules: 

1.  If  P(1)=0  then  F  =  0. 

2.  If  P(0)=0  then  F  =  global  mean  of  the  demand  series. 

3.  If  P( 1 | 1)  >=  P( 0 | 1)  then  F  =  average  of  the  previous  actual 
demand  since  the  last  zero  demand. 

4.  If  P(l|0)  >  P(ojo)  then  F  =  average  of  the  previous  actual 
demand  since  the  last  zero  demand. 

5.  If  none  of  the  above  are  true  then  F=0. 
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where:  F  =  QFD 

P(l)  =  probability  that  the  transformed  demand  equals  1 

P(0)  =  probability  that  the  transformed  demand  equals  0 

P(l|l)  =  probability  that  the  transformed  demand  equals  1 
given  that  the  previous  demand  equals  1 

P(0 | 1)  =  probability  that  the  transformed  demand  equals  0 
given  that  the  previous  demand  equals  1 

P(  1 1 0)  =  probability  that  the  transformed  demand  equals  1 
given  that  the  previous  demand  equals  0 

P(0 | 0)  =  probability  that  the  transformed  demand  equals  0 
given  that  the  previous  demand  equals  0 

F.  Combinations  of  models.  An  average  of  each  pair  of  forecasting 
techniques  was  used  as  a  forecast. 

VI.  EVALUATION  MEASURES 

The  criteria  used  for  evaluating  the  forecasting  model’s  performance,  or 
accuracy,  was  an  area  of  particular  concern  for  this  study.  The  decision 
should  come  from  the  manager  to  insure  agreement  with  DLA  policies  and 
goals  (20,17  p4,12  p32) .  Management's  view  of  what  is  most  important  with 
regard  to  forecast  accuracy  can  then  be  translated  into  appropriate  error 
computations.  In  contrast  to  the  forecast  accuracy  measures  that  are  used 
in  the  technique  selection  process  and  are  computed  over  the  testing  period 
(section  IV) ,  the  evaluation  measures,  described  here,  are  computed  from 
forecast  error  over  the  results  period.  In  other  words,  after  a  technique 
is  selected  to  forecast  an  item's  demand,  it  will  be  evaluated  by  its 
ability  to  forecast  the  future.  This  is  termed  the  forecasting  model's 
evaluation  and  encompasses  more  than  the  selected  technique's  forecasting 
accuracy;  but  also,  and  more  importantly,  the  process  by  which  the  model 
selects  that  technique. 

The  Modified  Index  of  Predictive  Efficiency  (MIPE)  is  a  relative  measure 
that  provides  an  item  by  item  comparison  of  SAMMS  forecast  to  the  model 
forecast.  It  is  computed  as  follows: 


MIPE  =  100  * 


(Es  -  iy 

*  (Es  +  Em) 


where:  Es  =  TAE  of  SAMMS'  forecast 


Efl,  =  TAE  of  model's  forecast 
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Although  there  are  some  obvious  difficulties  in  interpreting  the  MIPE,  it 
lus  two  characteristics  which  make  it  a  good  choice  at  this  time.  First, 
the  MIPE  provides  a  comparison  to  the  forecast  generated  by  SAMMS  on  an 
item  by  item  basis.  Thus  each  item's  forecasting  accuracy  has  the  same 
weight.  This  trait  was  desirable  to  prevent  forecasts  for  a  few  high-demand 
items  from  having  a  large  impact  on  the  results.  Second,  it  enabled  a  fair 
comparison  between  the  accuracy  of  the  model's  forecast  and  that  of  SAMMS. 
To  illustrate  these  robustful  qualities,  consider  the  following  example. 

1 .  Model ' s  forecast  error  =  3 
SAMMS'  forecast  error  =  1 

Percent  error  reduction  =  100  *  (1  -  3)/l  =  -200% 

MIPE  =  100* (1  -  3)  /  [ H* ( 1  +  3)]  =  -100% 

2.  Model's  forecast  error  =  1 
SAMMS'  forecast  error  -  3 

Percent  error  reduction  =  100  *  (3  -  l)/3  =  67% 

MIPE  =  100* (3  -  1)  /  [ :,* (3  +  1)]  =  100% 

As  this  example  illustrates,  SAMMS'  forecasts  are  favored  when  computing  a 
simple  percent  reduction.  When  the  model's  forecasting  error  was  two  units 
more  than  SAMMS',  its  performance  was  200  percent  less.  However  when  the 
model's  error  was  two  units  less,  its  performance  was  only  67  percent  more. 
By  using  the  MIPE  both  cases  yield  equal,  but  opposite,  results. 

Two  additional  evaluation  measures  were  also  used  as  forecasting  accuracy 
measures  and  were  discussed  section  IV. A.  The  MAD  was  used  for  selected 
analyses,  such  as  long-term  system  performance  (sections  IX. D. 2  and  IX. E) , 
because  of  its  computational  efficiency  over  the  MIPE.  The  TAE  is  used 
throughout,  but  primarily  as  a  raw  value  to  complement  relative  measures. 

VII.  DEMAND  BASE  DEVELOPMENT 

Items  included  for  this  study  and  their  respective  characteristics,  such  as 
procurement  cycle  period  and  lead  times,  were  taken  from  the  Supply  Control 
Files  in  second  quarter  FY  1989.  Demand  histories  were  constructed  by 
using  the  Supply  Control  Files  from  second  quarter  FY  1984  through  second 
quarter  FY  1989.  The  criteria  for  the  selection  of  items  included  the 
following. 

1.  Demand  supported  replenishment  items  (Item  Category  Code  "1") 

2.  Established  items  (Age  of  Item  Code  "E") 

3.  Stocked  items  (Supply  Status  Codes  other  than  "2,"  "3,"  ”9") 

4.  Quarterly  forecasted  items  (Forecast  Basis  Code  "2") 

5.  The  item  must  have  6  years  of  available  demand  history  where 
zero  demand  is  an  acceptable  history. 
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The  multiple  forecasting  model's  evaluation  focused  on  yearly  forecast 
accuracy.  The  length  of  the  forecast  horizon  requires  consideration  of  the 
amount  of  time  it  takes  to  procure  and  stock  an  item.  Given  that  a  forecast 
is  accurate  for  events  over  the  next  30  or  90  days,  the  resources  to  react 
to  that  forecast  may  not  be  available.  Table  1  represents  some  pertinent 
characteristics  of  the  items  selected  according  to  the  stated  criteria. 


TABLE  1 

PCRJLATION  ITEM  OlARACTERISTICS 


COMMODITY 

NUMBER 

OF  ITEMS 

AVG  PROCUREMEOT 
CYCLE  PERIOD 
(DAYS) 

AVG  TOTAL 
LEAD  TIME 
(DAYS) 

Construction 

80,668 

552 

213 

Electronics 

130,184 

744 

231 

General 

75,664 

746 

213 

Industrial 

215,370 

872 

272 

Medical 

9,771 

532 

251 

Three  samples  of  items  for  each  commodity  were  used  for  the  analysis. 
Because  of  the  larger  population  size  and  to  provide  roughly  similar  sample 
sizes  for  each  commodity,  the  percent  of  Industrial  items  sampled  is 
smaller.  The  following  is  a  list  of  the  samples  and  their  respective 
purposes.  The  population  percentage  for  Industrial  samples  is  contained  in 
parenthesis. 


TABLE  2 
ITEM  SAMPLES 


PORJLATION 

SAMPLE  PERCENT  RJRPOSE 


1 

2  (1) 

Selection  process  analysis 

2 

2  (1) 

Demand  base  analysis 

3 

10  (5) 

Validation  and  summary  statistics 
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VIII.  SAMMS  FORECAST 


To  evaluate  the  performance  of  the  multiple  forecasting  model,  comparisons 
were  made  to  the  accuracy  of  a  SAMMS  forecast.  SAMMS  forecast  is  a  double 
exponential  smoothing  technique  (see  section  V.C.2).  Total  demand,  which 
consisted  of  recurring  and  nonrecurring  demands,  was  used  for  the  forecast 
generation.  No  efforts  were  made  to  exclude  certain  types  of  demands. 
Smoothing  constants  were  initialized  with  a  four  quarter  average  and  an 
alpha  of  0.2  was  used  as  the  smoothing  parameter.  Comparisons  were  also 
made  using  a  SAMMS'  alpha  of  0.15;  however,  limited  sensitivity  analysis 
reveals  that  the  difference  in  total  performance  was  insignificant. 

The  recorded  QFDs  were  not  used  for  the  comparitive  analysis.  This  was 
necessary  since  at  times  great  disparity  exists  between  the  QFD  of  record 
and  the  demand  history.  This  difference  is  not  well  understood  and 
limited  analysis  suggests  that  this  results  from  SAMMS  single  and  double 
smoothed  quantities  that  are  either  revised  to  accommodate  competing 
supply  objectives  or  are  simply  in  error  (31) .  This  does  not  indicate 
that  any  alteration  of  these  quantities  has  a  negative  impact  on  supply 
management  goals;  but,  rather,  that  any  comparative  analysis  conducted 
using  recorded  QFDs  may  introduce  inadequacies  that  would  prevent  long-term 
predictability  of  forecasting  requirements. 

IX.  ANALYSIS 

A  general  form  of  the  forecasting  model  was  defined  in  Section  III.  The 
determination  of  the  best  combination  of  criteria  that  would  govern  the 
model's  selection  and  forecasting  process  is  a  complex  task  because  of  the 
many  alternatives  previously  presented.  For  this  reason,  detailed  testing 
of  the  model's  actual  performance  was  delayed  until  its  operation  was  more 
clearly  defined.  During  the  early  stages  of  the  project,  emphasis  was 
placed  on  the  development  of  point  estimates  to  include  measures  of  the 
average  error.  Subsequently,  these  average  performance  measures  were 
employed  to  expedite  model  development.  Once  defined,  more  rigorous 
performance  testing  could  be  conducted. 

The  first  task,  which  is  discussed  in  section  IX. A,  was  to  determine  how 
the  forecasting  model  would  select  the  best  forecasting  technique  for  an 
item.  Secondly,  and  with  that  selection  process  in  place,  which  subset  of 
forecasting  techniques  provides  the  best  results.  The  identification  of 
the  subset  of  techniques  is  presented  in  section  IX. B.  Remaining  areas  of 
analysis,  such  as  the  most  appropriate  demand  base  and  system  impact 
issues,  will  follow. 

A<  SELECTION  PROCESS.  The  selection  process  is  the  core  of  the 
multiple  forecasting  model.  It  defines  how  to  select  the  "best" 
forecasting  technique  from  all  available  forecasting  techniques.  This 
analysis  comprised  the  evaluation  of  various  types  of  forecast  application, 
error  measures  and  thresholds  for  bias  detection.  The  following  matrix 
depicts  the  various  alternatives  evaluated. 
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TABLE  3 

SELECTION  CRITERIA  ANALYSIS  MATRIX 


ERROR:  MSE  MAD  TAE 


BIAS  DETECTION: 

.0 

.2 

L 

.4 

.0 

.2 

.4 

.0 

.2 

.4 

FORECAST  APPLICATION 
One-step  ahead 

1 

1 

1 

1 

1 

1 

1 

Straight-1 ined 

■ 

•ORM2 

. 

MCE 

Long  Term 

1 

1 

■ 

1 

1 

1 

1 

As  illustrated,  there  were  27  alternatives  and,  for  each  alternative,  there 
were  four  methods  of  obtaining  a  yearly  forecast  from  the  selected 
technique  (see  Section  III.B).  By  comparing  the  MIPEs,  the  top  three 
alternatives,  listed  belcw,  were  identified  for  further  study. 

1.  Commodity:  General 

a.  ALTERNATIVE  1: 

Minimize  the  MSE  from  one-step  ahead  forecasts. 

Detect  bias  at  the  0.4  level. 

Obtain  an  annual  forecast  by  multiplying  the  QFD  by  four. 

b.  ALTERNATIVE  2: 

Minimize  the  MAD  from  one-step  ahead  forecasts. 

Detect  bias  at  the  0.2  level. 

Obtain  an  annual  forecast  by  summing  four  QFDs  based  on 

using  previous  forecasts  as  the  actual  demand. 

c.  ALTERNATIVE  3: 

Minimize  the  TAE  from  one-step  ahead  forecasts. 

Detect  bias  at  the  0.4  level. 

Obtain  an  annual  forecast  by  multiplying  the  QFD  by  four. 
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2.  Commodity:  Industrial 


a.  ALTERNATIVE  1: 

Minimize  the  MSE  from  one-step  ahead  forecasts. 

Detect  bias  at  the  0.4  level. 

Obtain  an  annual  forecast  by  multiplying  the  QFD  by  four. 

b.  ALTERNATIVE  2: 

Minimize  the  MSE  from  one-step  ahead  forecasts. 

Detect  bias  at  the  0.4  level. 

Obtain  an  annual  forecast  by  summing  four  QFDs  based  on 
using  previous  forecasts  as  the  actual  demand. 

c.  ALTERNATIVE  3: 

Minimize  the  TAE  from  one-step  ahead  forecasts. 

Detect  bias  at  the  0.4  level. 

Obtain  an  annual  forecast  by  multiplying  the  QFD  by  four. 


Again,  the  three  alternatives  for  each  commodity  presented  here  represent 
the  best  of  the  27  total  alternatives  for  each  commodity  that  was 
evaluated.  Although  the  actual  scores,  or  MIPEs,  for  each  alternative  are 
not  provided,  some  important  observations  will  be  noted  by  examining  the 
top  three  cases.  First,  similarity  exists  with  the  identified  alternatives 
for  each  commodity.  Alternatives  1  and  3  for  each  commodity  are  the  same. 
Second,  the  one-step-ahead  forecast  was  consistently  better  than  the  other 
methods  of  forecast  application.  This  was  a  criteria  in  each  alternative 
for  each  commodity.  Finally,  higher  bias  detection  thresholds  were  common 
for  these  alternatives. 

Each  alternative's  performance  may  vary  as  different  subsets  of  forecasting 
techniques  are  included.  Therefore,  each  alternative  was  put  through  the 
technique  elimination  process. 

B.  TECHNIQUE  ELIMINATION.  Contained  within  the  forecasting  model 
were  171  forecasting  techniques.  This  total  included  18  single  techniques 
and  all  possible  two-way  combinations  of  the  single  techniques.  For 
forecasting  each  item's  demand,  the  model  chooses  one  of  the  171  techniques 
based  on  the  selection  criteria. 

The  field  of  forecasting  techniques  needed  to  be  narrowed.  An  observation 
made  during  the  selection  process  analysis  suggests  that  some  of  the 
techniques  were  selected  as  the  best  based  on  previous  data  but  performed 
poorly  when  used  to  make  new  forecasts.  In  addition,  the  large  number  of 
techniques  greatly  increased  the  model's  computer  run  time.  Following,  is 
a  list  of  single  techniques  as  described  in  Section  V. 
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1.  Four  quarter  moving  average  (MEAN4) 

2.  Eight  quarter  moving  average  (MEAN8) 

3.  Lagged  moving  average  based  on  autocorrelations  (AOORR) 

4.  Single  exponential  smoothing  with  alpha=0.1  (SES.l) 

5.  Single  exponential  smoothing,  alpha=0.2  (SES.2) 

6.  Double  exponential  smoothing,  alpha=0.1  (DES.l) 

7.  Double  exponential  smoothing,  alpha=0 . 2  (DES.2) 

8.  SAMMS  Double  exponential  smoothing,  alpha=0 . 2  (DESnt) 

9.  Adaptive  response  single  exponential  smoothing  (ARRS2) 

10.  Adaptive  response  single  exponential  smoothing  with  a 

smoothed  alpha  (ARRSA) 

11.  Holt's  double  exponential  smoothing  (HOLTS) 

12.  Winter's  triple  exponential  smoothing,  alpha=0.1  (WINT1) 

13.  Winter's  triple  exponential  smoothing,  alpha=0.2  (WINT2) 

14.  Conditional  probability  model  based  on  clustering  (C1ROB) 

15.  Simple  linear  regression  (REGRS) 

16.  Nonlinear  regression  (NLREG) 

17.  Last  demand  (LASTD) 

18.  Year  ago  demand  (YRAGQ) 

For  each  run  of  the  forecasting  model  a  MIPE  was  computed  to  measure  single 
forecasting  technique  performance.  The  MIPEs  for  the  single  techniques 
were  then  aggregated  to  produce  an  overall  model  performance.  The  single 
technique  with  the  lowest  MIPE  was  eliminated,  one  at  a  time,  until  the 
overall  model  MIPE  showed  no  improvement.  As  this  implies,  the  forecasting 
model  was  run  approximately  15  times  for  each  of  the  top  three  alternatives 
that  were  identified  in  section  IX. A  for  each  commodity. 

This  was  not  an  exhaustive  search  for  the  right  techniques  for  each 
selection  alternative.  However,  it  did  provide  a  systematic  means  to 
review  the  performance  of  the  forecasting  techniques  and  provide  a 
respectable  set  of  techniques  which  ensured  the  carpi et ion  of  this  study.  A 
pertinent  statement  by  McCullagh  &  Nelder  is  given  (30  p23) : 


"Even  if  we  could  define  exactly  what  is 
meant  by  an  optimum  model  in  a  given  context, 
it  is  most  unlikely  that  the  data  would 
indicate  a  clear  winner  among  the  large  number 
of  carpeting  models.  We  must  anticipate  that, 
clustered  around  the  'best'  model  will  be  a  set 
of  alternatives  almost  as  good  and  not 
statistically  distinguishable." 


This  does  not  imply  that  the  search  for  better  techniques  should  abruptly 
end;  but  continuation  at  this  point  would  involve  a  considerable  amount  of 
analysis  with  minimal  additional  payoff. 
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The  amount  of  data  already  generated  as  a  result  of  approximately  90 
forecasting  model  applications  (15  runs  per  3  alternatives  per  2 
commodities)  has  become  somewhat  burdensome.  Although  this  data  is  readily 
available,  it  will  not  be  provided  here  for  clarity  and  report  size 
limitations.  Listed  below  are  the  most  appropriate  subsets  of  the  single 
forecasting  techniques  for  each  of  the  top  three  selection  alternatives 
defined  in  Section  IX. A. 


1.  Commodity:  General 

a. 

ALTERNATIVE  1: 

MEAN4 ,  MEAN8,  SES.l, 

REGRS, 

NLREG 

b. 

ALTERNATIVE  2: 

MEAN4,  MEAN8 ,  AOORR, 

SES.l, 

DES.l,  ARRSA,  NLREG 

c. 

ALTERNATIVE  3: 

MEAN8,  AOORR,  SES.l, 

NLREG 

2.  Commodity:  Industrial 


a. 

ALTERNATIVE  1: 

MEAN4 ,  MEAN8,  SES.l, 

NLREG 

b. 

ALTERNATIVE  2: 

MEAN4,  MEAN8,  YRAGO, 

SES.l,  DES.l 

c. 

ALTERNATIVE  3: 

MEAN8,  YRAGO,  SES.l, 

NLREG 

In  addition  to  the  sets  of  single  techniques  presented  above,  all  possible 
two-way  combinations  within  each  set  is  implicitly  included.  Again,  these 
sets  of  techniques  were  identified  as  appropriate  techniques  for  each 
alternative.  The  alternatives  now  encompass  not  only  the  method,  or 
criteria,  used  to  select  the  best  technique,  but  also,  the  set  of 
techniques  from  which  selections  can  be  made. 

An  important  similarity  between  each  alternative  for  each  commodity  is  that 
MEAN4,  MEAN8 ,  SES.l,  and  NLREG  were  typically  included.  So  regardless  of 
which  selection  alternative  is  employed,  roughly  the  same  subset  of 
forecasting  techniques  constituted  a  better  performer.  A  second 
observation  is  that  within  each  subset  is  a  varied  group  of  techniques. 
The  similarity  among  alternatives  and  the  variety  within  each  alternative 
lend  some  credibility  to  the  technique  elimination  process  employed  for 
this  study.  Summary  results  for  annual  forecast  accuracy  are  contained  in 
Table  4. 
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TABLE  4 

SELECTION  ALTERNATIVE  RESULTS 
ITEM  SAMPLE  3 


GENERAL  AVERAGE  AVERAGE 

ALTERNATIVE  MIPE{%)  TAE 


1  4  208 

2  4  219 

3  3  213 

SAMMS  N/A  219 


INDUSTRIAL  AVERAGE  AVERAGE 

ALTERNATIVE  MIPE(%)  TAE 


1  6  904 

2  3  943 

3  2  942 

SAMMS  N/A  957 


The  selection  procedure  defined  for  alternative  one  outperforms  the  others 
with  higher  predictive  efficiencies,  MIPEs  of  4  and  6  percent,  and  lower 
absolute  errors,  TAEs  of  208  and  904,  respectively  by  commodity. 
Intuitively,  one  selection  procedure  will  provide  better  results  fcr  each 
commodity.  This  idea  is  confirmed  by  the  similar  results  obtained  for  the 
General  and  Industrial  commodities. 

Realizing  that  different  supply  centers  manage  items  that  contain  different 
types  of  demand  patterns,  the  technique  elimination  was  continued.  The 
technique  elimination  procedure  for  Construction  yielded  the  same  set  of 
forecasting  techniques  as  Industrial  {MEAN4,  MEAN8,  SES.l,  NIPEG).  Analysis 
of  Medical  item  demands  yielded  a  slightly  different:  set  of  techniques  than 
those  listed  above;  however,  when  comparing  the  set  of  techniques  selected 
for  Medical  to  those  selected  for  General,  Industrial,  and  Construction  no 
measurable  difference  in  the  model's  performance  was  observed.  Based  on 
these  results,  the  same  set  of  techniques  were  applied  to  the  Electronics 
commodity. 

The  set  of  appropriate  techniques  has  not  been  examined  in  enough  detail  to 
say,  "Yes,  this  is  the  best  set  of  techniques  for  this  commodity."  However 
some  conclusions  can  be  drawn.  Smoothing  models  perform  better  at  lower 
alpha  levels.  Techniques  should  minimize  the  potential  for  volatile 
forecasts,  thereby  minimizing  the  loss  should  an  inappropriate  technique  be 
selected.  Some  diversification  must  exist  among  the  set  of  techniques. 
Seasonal  techniques  were  not  as  effective  and  is  most  likely  due  to  a 
yearly  forecast  horizon.  Finally,  continued  improvements  can  be  made  by 
ongoing  studies. 
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C.  DEMAND  BASE  ANALYSIS 

l.  Dampening.  Some  DLA  managed  items  have  a  tendency  for 
extreme  demands.  Regardless  of  the  magnitude,  these  can  skew  forecasts  and 
make  them  almost  meaningless.  An  attempt  was  made  to  smooth,  or  dampen, 
the  demands.  Dampening  is  defined  as  establishing  upper  and  lower  limits 
on  acceptable  demands.  If  a  time  period's  demand  is  beyond  the  set  limits, 
the  demand  is  altered  to  be  equal  to  the  limit.  Dampening  would  take  place 
within  the  forecasting  model  and  thus  not  artificially  change  the  demand  of 
record. 

Establishing  the  limits  is  an  extremely  difficult  task.  Care  must  be  taken 
to  include  real  changes  in  the  demand  structure  while  eliminating  the 
inpact  of  outliers.  Any  criteria  established  is  potentially  biased  and 
could  result  in  serious  departures  from  the  expected  value  of  demand. 
Limited  analysis  of  these  problems  is  continued  in  section  IX. G.  The 
limits  evaluated  were  multiples  of  the  standard  deviation  of  demand  such  as 
1,  2,  or  3  standard  deviations  from  the  average  item  demand.  This  criteria 
could  be  applied  evenly  to  all  items.  The  following  actual  demand  streams 
for  two  medical  items  will  serve  as  examples  for  further  discussion. 

ITEM  A: 


QUARTER 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10  11 

12 

ACTUAL 

0 

0 

12 

0 

36 

11 

369 

211 

331 

455  762 

890 

MODELS 

0 

0 

12 

0 

36 

11 

142 

142 

ITEM  B: 

QUARTER 

1 

2 

3 

4 

5 

6 

7 

8 

9 

ACTUAL  2364  2654  2415  39317  3456  3225  824  4108  5056  2326 

MODELS  2364  2654  2415  14537  3456  3225 


Actual  item  quarterly  demand  is  represented  by  the  "ACTUAL"  row.  Item 
demand  dampened  to  within  one  standard  deviation  of  the  mean,  142  for  item 
A  and  14,537  for  item  B,  is  represented  by  the  "MODELS"  row.  The 
forecasting  model's  yearly  forecast  would  be  for  the  last  four  quarters  of 
actual  demand.  Sixteen  quarters  of  data  were  available  for  each  item  but 
only  the  most  recent  is  sufficient  for  our  discussion.  For  both  items,  the 
four  quarter  moving  average  was  the  selected  technique. 

Item  A  is  an  example  of  when  not  to  dampen  the  demand  stream  and  was 
reflected  in  a  large  forecast  error.  Dampening  that  occurred  in  quarters 
seven  and  eight  prevented  the  forecasting  model  from  detecting  a  trend. 
Item  B  is  an  example  of  the  appropriate  use  of  dampening.  Dampening  that 
occurred  at  quarter  four  reduced  the  model's  forecasting  error  by  50 
percent. 
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To  compensate  for  both  types  of  demand  streams,  a  routine  was  developed  to 
permit  flexibility  in  the  dampening  process  by  attempting  to  identify  a 
changing  demand  mean.  In  particular,  the  latest  available  data  to  be  used 
in  a  forecast  was  not  dampened  if  more  than  one  quarter  was  beyond  the 
established  limit.  Therefore  for  the  first  item,  no  dampening  would  take 
place.  Various  levels  of  dampening  were  examined  and  the  results  for 
yearly  forecast  errors  are  shewn  in  Table  5. 


TABLE  5 

DEMAND  DAMPENING 
ITEM  SAMPLE  2 


GENERAL  DEGREE  OF 

DAMPENING 

AVERAGE 

MIPE(%) 

AVERAGE 

TAF 

UNDAMPENED 

0 

165 

3  STANDARD  DEVIATION 

1 

163 

2  STANDARD  DEVIATION 

4 

158 

1  STANDARD  DEVIATION 

7 

142 

SAMMS 

N/A 

166 

INDUSTRIAL  DEGREE  OF 
DAMPENING 

AVERAGE 

MIFE(%) 

AVERAGE 

TAE 

UNDAMPENED 

5 

1136 

3  STANDARD  DEVIATION 

6 

1125 

2  STANTARD  DEVIATION 

8 

1081 

1  STANDARD  DEVIATION 

11 

1073 

SAMMS 

N/A 

1200 

As  depicted,  the  model's  forecasting  accuracy  improved  dramatically  as  the 
demand  stream  was  dampened.  Dampening  not  only  improved  the  forecasts  but 
also  had  a  synergistic  effect  on  the  technique  selection  process.  By 
reducing  the  noise  the  model  could  more  appropriately  select  the  best 
forecasting  technique. 

2.  Recurring  and  nonrecurring  demand.  The  second  part  of  the 
demand  base  analysis  was  to  determine  the  types  of  demands  to  use  in  the 
forecasts.  There  were  two  components  of  demand  under  consideration: 
recurring  and  nonrecurring.  Initially  the  coefficient  of  variation  was 
computed  for  all  quarterly  forecasted  items.  As  shown  in  Appendix  A,  the 
demand  variability  was  reduced  by  adding  recurring  and  nonrecurring  (total) 
demand  quantities.  Even  when  only  one  quarter,  of  twenty,  experienced 
nonrecurring  demand  the  associated  variability  was  lower  for  total  demand 
than  recurring  demand  only. 
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TABLE  6 

NONRECURRING  DEMAND  ANALYSIS 
BY  SELECTION  ALTERNATIVE 
ITEM  SAMPLE  2 


GENERAL 

PERCENT  OF 
NONRECURRING 

AVERAGE 

MIPE(%) 

AVERAGE 

TAE 

100 

7 

142 

50 

11 

140 

0 

14 

141 

SAMMS 

N/A 

166 

INDUSTRIAL  PERCENT  OF 
NONRECURRING 

AVERAGE 

MIFE(%) 

AVERAGE 

TAE 

100 

11 

1073 

50 

12 

1065 

0 

10 

1092 

SAMMS 

N/A 

1200 

The  annual  forecast  errors  for  various  levels  of  nonrecurring  demand  are 
given  in  Table  6.  Seme  differences  were  evident  as  a  smaller  percentage  of 
nonrecurring  demand  was  applied  to  the  model's  forecasts.  However  the 
differences  for  each  commodity  provided  conflicting  results.  Moreover, 
there  may  be  little  consistency  in  labeling  each  demand  as  either  recurring 
or  nonrecurring.  Since  the  dampening  procedure  is  fairly  applied  to  all 
items  and  still  limits  the  effects  of  occasional  demand  spikes,  total 
demand  was  used  in  the  forecast  generation  process.  Ihe  demand  analysis  of 
the  remaining  commodities  suggests  similar  results  and  conclusions. 

To  summarize  the  demand  analysis,  below  are  the  model's  current  parameters 
and  performance  results  for  annual  forecast  errors  by  commodity. 

Current  model  parameters: 

Minimize  the  MSE  from  one-step  ahead  forecasts. 

Detect  bias  at  the  0.4  level. 

Obtain  a  annual  forecast  by  multiplying  the  QFD  by  four. 

Single  techniques  included:  MEAN4,  MEAN8,  SES.l,  NLRBG 
Technique  combinations  included:  all  possible  2-way 
combinations  of  the  single  techniques 
One  standard  deviation  demand  dampening 
Total  demand:  recurring  and  nonrecurring 
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TABLE  7 

MULTIPLE  FORECASTING  MODEL  SUMMARY  RESULTS 
ITEM  SAMPLE  3 


COMMODITY 

EIPE 

(%) 

AVERAGE 

TAE 

SAMMS  AVERAGE 
TAE 

%  TAE 
REDUCED 

Construction 

8 

219 

233 

6 

Electronics 

8 

129 

144 

10 

General 

10 

196 

219 

11 

Industrial 

10 

865 

957 

10 

Medical 

4 

482 

533 

10 

3.  TIMING  ANALYSIS.  Timing  analysis  deals  with  the  amount  of 
data  that  is  sufficient  to  obtain  improved  forecasts.  More  data  may  yield 
more  accurate  forecasts  or  may  introduce  extraneous  information  into  tne 
demand  stream,  slow  the  model's  forecasting  speed  and  create  larger  data 
files  to  be  stored.  Too  little  data  may  yield  meaningless  forecasts.  In 
addition  the  data  can  be  allocated  to  the  initialization  and  testing 
periods  in  various  ways.  Table  8  displays  the  timing  alternatives  tint 
were  evaluated. 


TABLE  8 

TIMING  ALTERNATIVE  DEFINITIONS 


ALTERNATIVE 

1 

2 

3 

INITIALIZATION  LENGTH 

12 

8 

8 

TESTING  LENGTH 

4 

4 

8 

QUARTERS  OF  DATA  REQUIRED 

16 

12 

16 

Alternative  one,  a  12  quarter  initialization  and  4  quarter  testing  period, 
was  used  until  the  forecasting  model  was  more  clearly  defined.  The  results 
for  alternative  one  were  previously  displayed  in  Table  7  as  summary 
results.  The  results  for  timing  alternatives  two  and  three  follow 
respectively  in  Tables  9  and  10. 
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TABLE  9 

TIMING  ALTERNATIVE  2  RESULTS 
ITEM  SAMPLE  3 


COMMODITY 

MI  PE 
(%) 

AVERAGE 

TAE 

SAMMS  AVERAGE 
TAE 

%  TAE 
REDUCED 

Construction 

6 

227 

233 

3 

Electronics 

7 

132 

144 

8 

General 

9 

205 

219 

6 

Industrial 

8 

873 

957 

9 

Medical 

4 

473 

533 

11 

COMMODITY 

TABLE  10 
TIMING  ALTERNATIVE 
ITEM  SAMPLE 

MI  PE  AVERAGE 

(%)  TAE 

3  RESULTS 

3 

SAMMS  AVERAGE 
TAE 

%  TAE 
REDUCED 

Construction 

9 

233 

4 

Electronics 

7 

139 

144 

3 

General 

9 

195 

219 

11 

Industrial 

9 

885 

957 

8 

Medical 

2 

527 

533 

1 

Alternative  one,  a  12  quarter  initialization  and  four  quarter  testing 
period,  yielded  more  accurate  forecasts.  This  conclusion  may  be  the  result 
of  bias  inherent  within  this  analysis  since  this  alternative  was  used  to 
establish  the  testing  period  parameters.  However,  the  following 
observations  were  made.  First,  consistency  in  forecasting  accuracy  across 
commodities  is  greater  with  the  first  alternative.  Secondly,  the  eight 
quarter  testing  period  used  for  alternative  three  may  not  be  responsive  to 
the  most  recent  demand  changes.  Finally,  alternative  two,  since  it 
requires  fewer  data  points,  may  be  applied  to  items  with  shorter  demand 
histories. 
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D.  OTHER  STATISTICS  AND  PERFORMANCE  RESULTS 


1.  Lead  time  error.  The  evaluation  of  the  forecasting  model's 
performance  had  been  based  on  a  yearly  forecasting  horizon.  A  year  was 
chosen  as  the  average  lead  time,  approximately  three  quarters,  plus  one 
quarter.  However  there  exists  a  wide  range  of  lead  times  for  DIA  items. 
The  lead  time  forecasting  errors  show  similar  increases  in  accuracy  with 
the  multiple  forecasting  model  and  are  depicted  in  Table  11. 


TABLE  11 

LEAD  TIME  FORECASTING  ERROR 
ITEM  SAMPLE  3 


COMMODITY  MIPE  AVERAGE  SAMMS  AVERAGE  %  TAE 

(%)  TAE  TAE  REDUCED 


Construction 

7 

173 

187 

7 

Electronics 

9 

111 

126 

12 

General 

10 

153 

173 

12 

Industrial 

11 

789 

882 

11 

Medical 

6 

316 

348 

9 

2.  Dynamic  vs.  Static.  Two  methods  of  model  implementation  were 
analyzed.  A  dynamic  forecasting  model  allows  an  item's  assigned 
forecasting  technique  to  change  for  each  quarterly  forecast.  The 
forecasting  model  would  evaluate  each  forecasting  technique  for  each  item 
every  quarter.  Concerns  regarding  this  mode  of  operation  include  potential 
problems  with  respect  to  computer  run  time  and  a  negative  effect  from  the 
churning  or  changing  of  forecasting  techniques  possibly  every  quarter. 

The  second  method  of  implementation  is  based  on  a  static  forecasting  model. 
This  model  would  select  the  best  technique  for  an  item  for  the  first 
quarterly  forecast  and  then  use  the  selected  technique  for  the  next  three 
quarters.  For  this  system,  a  multiple  forecasting  model  would  be  run  on  a 
yearly  basis. 

The  dynamic  vs.  static  analysis  necessitates  the  need  for  a  longer  term 
evaluation  and  therefore  attempts  were  made  to  replicate  system  performance 
over  a  specified  time  horizon.  The  previously  used  measures  of 
performance,  primarily  the  MIPE,  are  no  longer  appropriate  since  they  do 
not  lend,  computationally  and  interpret ively,  to  longer  term  evaluation. 
The  Mean  Absolute  Deviation  (MAD),  which  in  reality  is  an  average  TAE 
measure,  will  now  bo  used.  For  this  study,  the  yearly  MAD  is  defined  as 
the  average  absolute  error  of  four  yearly  forecasts  made  at  quarterly 
intervals  or  updated  quarterly.  The  quarterly  MAD  is  the  average  absolute 
error  of  four  quarterly  forecasts.  Tables  12  and  13  contain  results  for 
dynamic  and  static  forecasting  models. 
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TABLE  12 

DYNAMIC  SYSTEM  PERFORMANCE 
ITEM  SAMPLE  3 


YEARLY  %  IMPROVEMENT  QUARTER  %  IMPROVEMENT 


COMMODITY 

MAD 

OVER  SAMMS 

MAD 

OVER  SAMMS 

Construction 

254 

4 

79 

5 

Electronics 

116 

18 

43 

12 

General 

167 

14 

59 

9 

Industrial 

805 

9 

268 

7 

Medical 

414 

6 

118 

7 

TOTALS 

346 

9 

116 

8 

TABLE  13 

STATIC  SYSTEM  PERFORMANCE 
ITEM  SAMPLE  3 


COMMODITY 

YEARLY 

MAD 

%  IMPROVEMENT 
OVER  SAMMS 

QUARTER 

MAD 

% IMPROVEMENT 
OVER  SAMMS 

Construction 

254 

4 

79 

5 

Electronics 

119 

16 

43 

12 

General 

170 

12 

61 

6 

Industrial 

S45 

4 

279 

3 

Medical 

426 

3 

120 

6 

TOTALS 

359 

6 

119 

5 

The  dynamic  system  showed  more  improvement  in  all  commodities  except 
construction,  where  the  overall  results  were  identical.  This  analysis 
suggests  that  the  dynamic  system  is  more  effective.  It  allows  for  some 
changes  in  the  demand  structure  and  the  effect  of  selecting  an 
inappropriate  technigue  is  not  perpetuated  for  4  guarters  as  in  the  static 
system. 


One  of  the  major  concerns  with  the  dynamic  model  is  the  potential  for 
changing  the  forecasting  techniques  every  quarter.  This  concern  relates  to 
the  inherent  assumptions  within  this  type  of  multiple  forecasting  model. 
The  assumptions  are  restated  below. 

1.  One  forecasting  technique  will  be  the  best  for  a  given  item. 

2.  The  technique  that  worked  best  in  the  past  will  work  best  in 
the  future. 
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In  other  words,  if  a  technique  is  the  best  this  quarter,  one  would  expect 
it  to  be  the  best  the  next  quarter,  and  if  its  not  the  best  next  quarter, 
then  maybe  its  not  the  best  for  that  item.  The  later  implies  that  the 
demand  stream  does  not  contain  any  definite  time  period  from  which  the 
multiple  forecasting  model  can  select  the  best  forecasting  technique.  Is 
this  typical?  Since  the  two  assumptions  are  the  basic  principles  upon 
which  the  forecasting  model  was  constructed,  an  effort  should  be  made  to 
justify  or  validate  their  reasonableness  and  hence  the  integrity  of  the 
forecasting  model  itself. 

The  examination  of  this  question  focused  on  looking  for  occurrences  of  tho 
same  technique  being  selected  frequently  within  some  consecutive  time 
frame.  These  occurrences  could  then  be  compared  to  a  hypothetical  case  in 
which  forecasting  techniques  are  selected  at  random.  This  comparative 
analysis  gives  rise  to  two  frequency  distributions:  observed  vs.  random  or 
expected.  The  idea  would  be  to  conclude  that  the  observed  frequencies 
could  not  have  come  from  completely  random  technique  selections,  and 
therefore,  there  exist  some  time  periods  from  which  techniques  can  be 
selected.  More  simply,  the  technique  worked  best  in  the  past  so  it  is 
likely  to  work  best  in  the  future. 

Selected  forecasting  techniques  for  each  item  were  gathered  over  a  four 
quarter  time  horizon.  The  following  definitions  apply  to  this  discussion 
and  are  referred  to  as  levels  of  continuity.  "All  different"  is  used  1  r 
those  items  for  which  each  of  four  consecutively  selected  techniques  is 
different.  "One  pair"  is  used  for  those  items  in  which  one  pair  of  the 
four  selections  was  the  same  technique.  Similar  definitions  apply  to  "Two 
pair,"  "Three-of-a-kind, "  and  "Four-of-a-kind. "  Overall,  there  are  ten 
possible  techniques,  with  replacement,  (four  single  and  the  six  2-way 
combinations  of  the  single  techniques)  from  which  the  four  techniques  could 
be  selected. 

The  observed  frequencies  for  each  level  of  continuity  is  now  compared  to 
random  selections  of  four  techniques.  Employing  the  use  of  combinatorics 
one  can  obtain  the  expected  number  of  times,  under  the  random  case,  that 
each  level  of  continuity  would  be  observed. 


TABLE  14 

CONTINUITY  OF  TECHNIQUE  SELECTION  FROCESS 
ITEM  SAMPLE  3 


FREQUENCIES  FREQUENCIES 

OBSERVED  WITH  EXPECTED  FROM 

MODEL'S  SELECTIONS  RANDOM  SELECTIONS 


All  different 

4126 

20L87.2 

One  pair 

15160 

17303.3 

Two  pair 

4481 

1081.5 

Three-of-a-kind 

10887 

1441.9 

Four-of-a-kind 

5400 

40.1 

TOTALS 

40054 

40054.0 
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It  should  be  noted  that  the  expected  values  are  based  on  the  assumption 
that  each  technique  is  equally  likely  to  be  selected.  Moreover,  large 
sample  sizes  do  not  lend  themselves  to  classical  statistical  analysis! 
For  large  samples,  classical  analysis  typically  leads  to  hypothesis 
rejection  even  though  the  distributions  may  be  roughly  similar  (32  p20) . 
For  these  two  reasons,  a  classical  goodness-of-f it  test  would  be 
inappropriate.  However,  high  frequencies  are  associated  with  the  higher 
levels  of  continuity,  such  as  three  and  four-of-a-kind,  when  compared  to 
their  expected  values.  Also,  low  frequency  is  observed  for  the  lowest 
level  of  continuity.  The  forecasting  model  appears  to  be  more  likely  to 
select  the  same  forecasting  technique  that  was  previously  chosen  and  thus 
some  continuity  exists  from  quarter  to  quarter. 

Comparisons  between  the  two  distributions  indicate  that  the  selection 
process  is  not  entirely  random.  This  does  not  imply,  necessarily,  that  a 
non-random  process  is  better  with  respect  to  forecasting  accuracy.  How,  is 
continuity  associated  with  increased  forecasting  accuracy?  Indeed,  as 
Table  15  illustrates,  forecasting  accuracy  improves  to  a  19  percent  error 
reduction  as  the  level  of  continuity  increases. 


TABLE  15 

TECHNIQUE  SELECTION  PERFORMANCE 
BY  CONTINUITY  LEVEL 
ITEM  SAMPLE  3 


YEARLY 

MAD 

%  IMPROVEMENT 
OVER  SAMMS 

QUARIER 

MAD 

%IMPROVEMENT 
OVER  SAMMS 

All  different 

413 

4 

136 

2 

One  pair 

397 

7 

130 

5 

LVo  pair 

318 

12 

109 

11 

Three-of-a-kind 

337 

12 

117 

9 

Four-of-a-kind 

193 

19 

63 

17 

TOTALS 

346 

9 

116 

8 

The  last  issue  concerning  a  dynamic  model  is  computer  time.  To  get  a 
better  time  estimate,  the  forecasting  model  was  recoded  to  approximate  its 
use  in  the  production  environment.  This  was  accomplished  by  stripping  out 
some  of  the  features  used  for  assessment  that  allowed  various  parameter 
changes  to  be  made  quickly.  In  essence  the  model  was  "hard-wired"  but 
still  in  a  modular  form.  The  approximate  computer  time  is  10  CRJ  minutes 
per  100,000  line  items.  So  with  Defense  General  Supply  Center's  Amdahl  V8, 
the  model  can  forecast  roughly  100,000  items  in  about  ten  clock  minutes. 

3.  Item  by  item.  It  was  shown  that  average  forecasting  error 
decreased  with  multiple  forecasting.  One  may  ask  if  this  lower  error  can 
be  observed,  significantly,  on  an  item  by  item  basis.  The  purpose  of  using 
the  WIPE  as  a  performance  measure  was  an  attempt  to  prevent  the  errors  of  a 
few  items  from  dominating  the  results,  and  hence,  the  evaluation.  The 
significance  of  the  lower  MADs  produced  by  the  forecasting  model  is  next 
examined . 


28 


For  this  analysis,  a  nonparametric  sign  test  was  used.  This  test,  or  a 
variation  of  it,  is  typically  used  to  compare  the  means  of  two 
distributions  when  the  distributions  are  not  well  understood.  Clearly,  the 
model  MADs,  across  all  sample  items,  does  not  constitute  a  sample  from  a 
known  distribution.  Similarly,  SAMMS  MADs  are  not  observations  from  a 
known  distribution. 

The  testing  procedure  follows.  First,  count  the  number  of  items  where  a 
smaller  error  was  observed  as  the  result  of  the  model's  forecasts.  Second, 
count  the  total  number  of  model  forecasts  compared  to  SAMMS  forecasts. 
Items  with  equal  forecasting  errors  were  excluded  from  these  frequencies 
which  are  displayed  in  Table  16. 


TABLE  16 

FREQUENCIES  OF  IMFROVED  FORECASTS 
ITEM  SAMPLE  3 


COMMODITY 

FREQUENCIES 
FOR  LOWER 
OBSERVED  MAD 

PERCENT 

OF 

TOTAL 

TOTAL 
NUMBER  OF 
OBSERVATIONS 

Construction 

4698 

.627 

7495 

Electronics 

7013 

.644 

10887 

General 

3911 

.652 

5996 

Industrial 

6842 

.674 

10151 

Medical 

408 

.608 

671 

TOTALS 

22872 

.650 

35200 

If  forecasting  error  associated  with  the  forecasting  model  and  SAMMS  are 
statistically  indistinguishable,  one  would  expect  each  to  provide  the  best 
forecast  50  percent  of  the  time.  The  sign  test  indicates  that  the 
probability  of  the  observed  frequencies  occurring  randomly  is  extremely 
small.  Thus  the  errors  produced  by  the  forecasting  model  are  smaller  than 
those  produced  by  SAMMS'  forecasts.  However,  here  again,  care  must  be 
taken  when  using  classical  statistics  or  interpreting  confidence  levels 
derived  from  large  samples.  A  more  meaningful  result  is  that  an  estimated 
65  percent  of  the  time  forecasts  produced  by  the  multiple  forecasting  model 
will  be  more  accurate  than  a  SAMMS  forecast.  Cue  to  the  stochastic  nature 
of  demands,  more  accurate  forecasts  cannot  be  obtained  for  every'  forecast 
period  for  every  item. 

E.  SYSTEM  IMPACT.  The  Materiel  Readiness  System's  Performance 
Projection  Model  (PERMES)  was  used  to  evaluate  the  forecasting  model's 
impact  on  supply  availability  and  stock  fund  investment.  PERMES,  by  using 
inventory  control  theory  and  historical  demand  variance,  computes  future 
supply  performance  and  expected  requirements  (26) .  The  MAD  provides 
the  historical  demand  variance  estimate  for  DLA's  inventory  system. 
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MADs  for  the  model  and  SAMMS  were  computed  from  four  quarterly  forecasts 
for  the  population  of  items  evaluated  in  this  study.  Both  sets  of  computed 
MADs  and  new  forecasts  were  then  overlaid  onto  the  PERMES  data  files. 
Computed  MADs,  used  as  PERMES  input  for  the  population  of  items  in  this 
study,  are  displayed  in  Table  17.  This  table  also  provides  the  percent 
improvement  over  SAMMS  for  these  items  and  thus  the  best  estimate  for 
quarterly  MAD  reduction  associated  with  the  forecasting  model. 

TABLE  17 

PERMES  INPUT:  MEAN  ABSOLUTE  DEVIATION 
PORJLATION 


QUARTERLY  MAD 

NUMBER  OF  r -  AVERAGE - i  %  IMPROVEMENT 

COMMODITY  ITEMS  \  SAMMS  MODEL  |  OVER  SAMMS 


Construction 

80668 

83 

79 

4 

Electronics 

130184 

35 

32 

9 

General 

75664 

58 

52 

10 

Industrial 

215370 

298 

270 

9 

Medical 

9771 

97 

93 

4 

Total 

511657 

158 

144 

9 

Preliminary  PERMES  runs  were  made  to  determine  the  system  constants  and  the 
number  of  requisitions.  The  system  constant  is  the  sum  for  each  commodity 
of  the  product  of  each  item's  unit  price  and  lead  time  MAD.  Hence  it  is  a 
fixed  quantity  determined  by  forecasting  accuracy. 

The  number  of  requisitions  was  used  to  estimate  the  Beta  goals  for 
established  backorder  lines.  The  Beta  or  established  lines  backordered 
goal  is  indirectly  used  by  SAMMS  as  a  supply  availability  goal  and  SAMMS 
computes  the  associated  safety  levels  required  to  achieve  this  goal. 
Attempts  were  made  to  obtain  current  goals  that  are  established  by  supply 
center.  However  these  values  apply  to  all  items  managed  by  each  respective 
center,  not  just  those  included  in  this  study.  The  PERMES  preliminary 
analysis  data  is  shown  in  Table  18. 

TABLE  18 

PRELIMINARY  PERMES  STATISTICS 
POPULATION 

CURRENT  SYSTEM  CONSTANT 

ANNUAL  SAFETY  I -  OOMFUTED  - 1 


COMMODITY 

REQUISITIONS 

LEVEL ($T) 

MODEL 

SAMMS 

Construction 

2995972 

168825 

173267792 

186084432 

Electronics 

1853896 

51074 

94713552 

102558000 

General 

806847 

107470 

119592992 

129715088 

Industrial 

4475067 

129698 

182405952 

198693840 

Medical 

309906 

21250 

15710362 

16861232 

Total 

478317 
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Beta  goals  and  the  system  constants  provide  the  needed  input  for  complete 
analysis  runs.  System  constants  for  each  scenario,  forecasting  model  and 
SAMMS,  were  taken  directly  from  Table  18.  However,  to  get  a  better 
understanding  of  the  importance  of  the  Beta  goals  in  supply  performance  and 
safety  levels,  the  sensitivity  of  safety  level  dollars  to  Beta  changes  was 
analyzed.  PERMES  analysis  runs  were  made  for  Beta  goals  of  6,  3,  1.5,  and 
1  percent  of  the  number  of  annual  requisitions.  A  Beta  goal  of  6  percent 
would  imply  an  annual  fill  rate  of  94  percent.  This  fill  rate  appears 
unreasonably  high,  but  PERMES  is  a  projection  model  which  may  over  estimate 
availability  due  to  its  method  of  stochastical  demand  generation.  More 
importantly,  however,  is  that  current  safety  levels,  which  are  depicted  m 
Table  18,  are  more  accurately  reflected  in  the  higher  Beta  goals. 

The  same  Beta  values  that  were  used  to  approximate  the  safety  levels 
associated  with  the  model's  forecasts  were  also  used  for  the  levels 
associated  with  SAMMS'  forecasts.  The  implication  of  this  analysis  is  that 
one  would  expect  PERMES  to  compute  approximately  the  same  projected  supply 
availabilities  for  both  scenarios.  However  a  slight  improvement  in  supply 
availability  was  observed  with  model  MADs  and  forecasts.  Graphs  depicting 
the  impact  on  supply  availability  and  safety  level  of  the  various  Beta 
goals  are  contained  in  Appendix  B.  Table  19  represents  tne  estimated 
safety  level  reduction,  in  thousands,  based  on  a  Beta  goal  that  most 
closely  reflects  the  current  safety  levels. 

TABLE  19 

PERMES  SAFETY  LEVEL  ANALYSIS 
POPULATION 


BETA  GOAL  AS  SAFETY  LEVEL  SAFETY 

%  ANNUAL  | — <30MPUTED($T) - 1  LEVEL 

COMMODITY  REQUISITIONS  MODEL  SAMMS  REDUCED ($T) 


Construct ion 

1.0 

105232 

115916 

10684 

Electronics 

1.5 

50160 

57328 

7168 

General 

1.0 

84524 

94673 

10149 

Industrial 

1.0 

107050 

120273 

13223 

Medical 

1.0 

8774 

9463 

689 

Estimated  Savings  41913 


The  estimated  one-time  savings  in  safety  level  reduction  is  $42  million. 
Results  indicate  that  this  reduction  can  be  accomplished  at  no  sacrifice  to 
supply  availability. 

The  estimated  savings  compares  favorably  with  the  assumption  that  for  each 
1  percent  reduction  in  MAD  there  is  a  corresponding  1  percent  reduction  in 
safety  level.  Employing  this  method,  the  overall  9  percent  MAD  reduction 
given  by  the  forecasting  model  (see  Table  17)  times  the  total  current 
safety  level  of  $478,317,000  (see  Table  18)  for  these  items  yields  an 
estimated  savings  of  $43  million.  This  is  indeed  a  comforting  result 
although  somewhat  surprising.  The  method  for  estimating  savings,  described 
here,  was  successfully  used  for  a  previous  DLA  demand  forecasting  study 
(20). 
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P.  MONTHLY  FORECASTING.  An  issue  of  concern  by  the  Requirements 
Forecasts  Working  Group  for  implementation  is  with  the  frequency  of 
forecasting.  Monthly  forecasting  is  taken  to  be  a  forecast  generated  from 
monthly  demands.  The  medical  commodity  was  selected  for  this  analysis  for 
the  following  reasons.  First  in  comparison  across  commodities,  medical 
items  typically  showed  less  improvement  over  SAMMS'  forecasts  (see  Table 
17) .  Second,  there  appears  to  be  more  items  with  rapidly  changing  demands 
which  suggests  that  items  are  phased  in  and  out  of  use  more  rapidly. 
Finally  this  commodity  is  associated  with  shorter  procurement  cycles  (see 
Table  1)  which  would  imply  more  frequent  buying  decisions.  Together  these 
observations  suggest  a  greater  urgency  for  monthly  forecasts. 

Two  scenarios  were  evaluated.  First,  sixteen  quarterly  demand  observations 
were  multiple  forecasting  model  inputs.  The  forecasting  model  estimated 
the  future  annual  requirements  by  multiplying  the  quarterly  forecast  by 
four.  Secondly,  sixteen  monthly  demands  were  used  as  inputs  and  future 
annual  requirements  were  obtained  by  multiplying  the  monthly  forecast  by 
twelve.  The  premise  of  this  analysis  was  that  the  monthly  forecast  would  be 
more  responsive  to  the  most  recent  changes  in  the  demand  stream. 

Again  because  of  the  unknown  underlying  distributions,  a  nonparametric  sign 
test  was  used.  First,  count  the  number  of  items  where  a  lower  annual  error 
was  observed  as  the  result  of  model  forecasts  based  on  quarterly  demands. 
Second,  count  the  total  number  of  observations  when  model  forecasts  based 
on  quarterly  demands  compared  to  model  forecasts  based  on  monthly  demands. 
Items  that  had  equal  forecasting  errors  were  excluded  from  the  frequencies 
displayed  in  Table  20. 


TABLE  20 

FREQUENCY  of  improved  forecasts  with  quarterly  demands 

SAMPLE  3 


NUMBER  OF  TIMES 
A  LOWER  ERROR 
OOMMODIIY  WAS  OBSERVED 


PERCENT  TOTAL 
OF  NUMBER  OF 

TOTAL  OBSERVATIONS 


Medical  427  .593  720 


Ihe  nonparametric  test  would  indicate  that  the  probability  of  the  observed 
frequencies  occurring  randomly  is  very  small.  In  other  words,  a 
probability  greater  than  50  percent  exists  for  obtaining  more  accurate 
forecasts  with  quarterly  demands.  IXie  to  the  large  sample  size  and  the  use 
of  classical  statistical  procedures,  this  does  not  imply  a  great  deal  of 
information.  A  more  meaningful  conclusion  is  that  annual  forecasts  with 
greater  accuracy  were  obtained  approximately  60  percent  of  the  time  when 
the  forecast  for  annual  requirements  were  based  on  quarterly  demands.  This 
should  be  viewed  as  the  approximate  probability  of  obtaining  a  more 
accurate  forecast.  A  7  percent  improvement  in  absolute  annual  forecasting 
error  was  also  observed.  Most  likely  this  result  is  explained  by  greater 
monthly  demand  instability  which  creates  more  instability  in  long-term 
forecasts.  Similar  conclusions  were  drawn  under  a  previous  DIA  study  (20) . 
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G.  FOLLOW-ON  ANALYSIS.  In  this  follow-on  analysis  two  questions 
were  addressed  with  respect  to  the  multiple  forecasting  model's 
performance.  This  analysis  is  critical  due  to  certain  weaknesses  within 
the  forecasting  model.  In  particular,  apparently  erratic  demands  cannot  be 
explained,  and  must  therefore,  be  dampened  to  downplay  their  impact  for 
forecasting  future  requirements.  From  a  statistical  sense  this  approach  is 
somewhat  naive  in  that  inherent  bias  may  be  incorporated  within  the  demand 
forecast  or  the  demand  variance  estimator,  the  MAD.  Are  the  forecasts 
biased?  Is  the  reduced  MAD,  which  is  a  by-product  of  the  model's  forecast, 
an  appropriate  estimator  for  demand  variance? 

1.  Bias.  If  the  forecasting  model  produces  biased  forecasts 
then  these  forecasts  would  consistently  under,  or  consistently  over, 
estimate  future  demands.  The  number  of  times  the  model  under  and  over 
forecasts  annual  demand  is  compared  to  the  number  of  times  SAMMS  under  and 
over  forecasts  annual  demand .  Items  with  the  minimum  forecast  of  one  and 
zero  demand  were  not  included  in  the  fol lowing  table. 


TABLE  21 
FORECASTING  BIAS 
SAMPLE  3 

GENERAL  1 -  NUMBER  - 1  I - PERCENT 


COMMODITY 

UNDER 

OVER 

UNDER 

OVER 

MODEL 

3256 

4056 

44.5 

55.5 

SAMMS 

2933 

4390 

40.1 

59.9 

A  tendency  exists  for  over-forecasting  item  demand.  This  tendency  is 
slightly  diminished  for  the  model's  forecasts.  One  explanation  is  that  low 
demand  items  tend  to  limit  the  potential  for  obtaining  a  smaller  forecast 
regardless  of  the  forecasting  methodology  employed. 


2.  Standard  Deviation  Estimator.  The  MAD  plays  an  important 
role  in  DLA's  inventory  system  as  a  demand  standard  deviation  estimator. 
Based  on  theoretical  work  contingent  upon  an  exponential  forecast  (14 
p37-4 1 ,  25  P282-287,  28,  29  pl4-20),  the  error  measure  is  shown  by 
simulated  data  to  be  appropriate  for  some  demand  distributions,  such  as  the 
normal  distribution.  The  general  approximation  follows. 

a  =  /3  *  MAD 

where:  a  ~  standard  deviation  of  demand 
fj  -----  scaling  factor 

The  scaling  factor  is  considered  to  be  1.25  for  normally  distributed 
demands  and  is  the  value  used  by  DLA .  Here,  the  MAD  is  a  single 
exponential  smoothed  value. 
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Under  the  normally  distributed  demand  assumption  and  using  the  forecast  as 
the  expected  value  of  demand  the  fol lowing  is  derived: 


Let  ,  —  ,  Xn  be  n  independent  observations,  or  item  demand 
quantities  for  n  items,  each  from  a  normal  distribution  with 
=  expected  value  of  X^,  =  standard  deviation  of 

X^,  and  i=l,  . . . ,  n.  Then 

Pr (  I  XA  -  I  <  aoi)=  pa 

where:  a  -  a  scaler  such  as  1,2,3,  or  4 

pa  =  a  constant  hat  is  independent  of 
the  item's  .amand  distribution's 
parameters 


Based  on  this  result  one  can  compute  the  expected  number  of  observations 
for  various  values  of  "a".  Observed  frequencies  were  tabulated  by  using 
1 . 25*MAD  as  the  demand  standard  deviation  estimate,  X-^  represents  the 
actual  demand  for  item  i,  and  is  the  forecast  or  expected  value  of 
demand  for  that  item. 

More  simply,  this  implies  counting  the  number  of  items  which  experienced 
demand  less  than  three  standard  deviations  below  the  forecast,  between  two 
and  three  standard  deviations  below  the  forecast,  between  one  and  two 
standard  deviations  below,  and  so  on.  Each  freqency  category  is  contained 
in  Table  22.  Negative  "a"  values  indicate  demands  below  the  forecast  while 
positive  values  indicate  those  above. 


TABLE  22 

DEMAND  DISTRIBUTION 
MAD  AS  STANDARD  DEVIATION  ESTIMATOR 
GENERAL  COMMODITY 

MODEL  MAD  SAMMS  MAD 

VALUE  OF  "a"  NUMBER  NUMBER  NUMBER 


MIN 

MAX 

OBSERVED 

OBSERVED 

EXPECTED 

-00 

-3 

6 

2 

9.7 

-3 

-2 

23 

28 

161.1 

-2 

-1 

338 

519 

1018.4 

-1 

1 

6096 

5971 

5115.4 

1 

2 

486 

453 

1018.4 

2 

3 

197 

209 

161.1 

3 

00 

348 

312 

9.7 

7494 

7494 

7493.8 

34 


There  are  a  few  features  that  deserve  to  be  mentioned  when  comparing  the 
observed  and  expected  frequency  distributions.  It  appears  that  for  a 
larger  number  of  times,  demands  are  observed  that  are  within  one  standard 
deviation  of  the  forecast,  or  mean.  Second  the  lower  half  of  the 
distribution  is  observed  with  smaller  frequencies  than  the  upper  half. 
Finally  the  general  shape  of  the  observed  and  expected  distributions  is 
similar.  Although  this  test  does  not  indicate  when  the  MAD  is  a  good 
estimate  for  the  standard  deviation,  some  conclusions  can  be  drawn. 

If  the  two  observed  distributions  are  really  different  from  the  expected 
distribution  then  the  potential  exists  for  two  problems  to  emerge.  Firse, 
the  MAD  may  not  be  an  appropriate  demand  variance  estimator.  Second,  the 
normality  assumption  does  not  hold. 

Since  the  observed  and  expected  distributions  are  somewhat  different, 
especially  in  the  tails,  the  MAD  resulting  from  the  multiple  forecasting 
model,  may  not  be  a  valid  estimator.  Similarly,  this  reclt  can  be  applied 
to  the  MAD  that  is  a  by-product  of  SAMMS'  forecasts.  Th  •  core,  most  of 
the  problems  associated  with  the  MAD  must  be  in  the  assumption  of  normally 
distributed  demands.  The  normal  assumption  cannot  be  unconditionally  used 
for  all  demand  replenishment  items.  Items  with  small  demand  frequencies 
and  quantities  are  most  likely  contributing  to  this  problem. 

X.  CONCLUSIONS 

A  multiple  forecasting  model  that  optimally  selects  a  forecasting 
technique,  such  as  the  one  presented  here,  works  for  DLA  items.  When 
long-term  forecasting  accuracy  is  desired,  the  best  technique,  of  all 
unbiased  techniques,  should  have  the  smallest  MSE  computed  from  four 
one-step  ahead  forecasts.  This  criteria  minimizes  the  potential  for  larger 
forecasting  errors  and  becomes  critical  when  the  large  forecasting  error  is 
multiplied  to  obtain  lead  time  demand. 

The  method  for  demand  dampening  applied  in  this  study  greatly  improved  the 
model's  forecasting  performance.  This  procedure  not  only  improves  the 
accuracy  of  the  selected  technique,  but  also  increases  the  likelihood  of 
selecting  a  more  appropriate  technique. 

Use  of  nonrecurring  demands  gave  more  conflicting  results.  There  is  some 
indication  that  applying  smaller  percentages  of  nonrecurring  demand  to  the 
forecast  can  improve  accuracy.  However  this  demand  classification  is  most 
likely  not  applied  evenly  and  its  removal  from  forecast  generation  may 
inherently  bias  the  forecasts. 

Forecasting  techniques  to  be  included  in  a  multiple  forecasting  model 
should  minimize  the  potential  for  an  extreme  forecast.  Moving  averages, 
single  exponential  smoothing,  and  regression  techniques  are  appropriate. 
Some  technique  diversity  exists;  however,  the  capacity  for  over-reacting  to 
one-time  or  rare  demand  spikes  is  minimal. 
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For  the  QFD  items  studied,  forecasting  error  can  be  reduced  by  9  percent 
with  a  multiple  forecasting  model.  Increased  forecasting  accuracy  has  an 
impact  on  safety  levels.  Without  decreasing  supply  availability,  the 
potential  one-time  safety  level  reduction  is  $42  million. 

Implementation  of  a  multiple  forecasting  system  should  center  on  a  model 
that  is  run  dynamically.  Each  quarter  a  new  technique  assignment  is  made. 
Associated  with  a  dynamic  system  are  smaller  forecasting  errors  and  high 
probabilities  of  forecasting  with  a  technique  selected  in  a  previous 
quarter. 

In  addition  the  forecasts  should  be  based  on  quarterly  demands.  Quarterly 
demand  observations  provide  more  stable  long-term  forecasts  and  increase 
the  likelihood  for  an  improved  forecast.  Appendix  C  contains  the  data 
requirements  of  a  multiple  forecasting  model. 

Follow-on  analysis  reveals  some  indication  of  possible  shortcomings 
associated  with  the  MAD  under  a  multiple  forecasting  system. 

XI.  RECOMMENDATIONS  AND  SUGGESTIONS  FOR  FURTHER  RESEARCH 

Specific  recommendations  stemming  from  this  project  are  the  following: 

►  A  multiple  forecasting  model,  as  defined  by  the  criteria  developed 
in  this  study,  should  be  implemented  to  provide  demand  forecasts  for  DLA 
items.  This  recommendation  applies  only  to  those  QFD  items  with  sufficient 
demand  histories  since  these  items  constituted  our  test  base. 

►  Items  with  shorter  demand  histories,  less  then  16  quarters,  can 
use  the  SAMMS'  forecasting  method  which  would  have  the  additional  benefit 
of  establishing  the  MAD  quantity. 

►  The  implementation  should  be  conducted  in  stages  with  appropriate 
evaluation  preceeding  each  stage. 

►►  Develop  working  prototype  at  a  hardware  center.  DORO  will  be 
available  for  technical  assistance  and  support. 

►►  Concurrent  and  subsequent  to  the  prototyping  efforts  the 
following  areas  should  be  addressed  by  the  RFWG: 

►►►  DLA  demand  forecasts  are  and  will  be  inaccurate,  as 
suggested  by  MADs  greater  than  QFDs.  How  do  we  best  cope  with  inaccurate 
forecasts?  How  do  we  identify  which  items  can  be  managed  more  effectively 
as  non-demand  based,  and  hence,  not  forecasted? 

►►►  This  study  has  focused  on  a  multiple  technique  approach 
to  forecasting  item  demands.  However  the  techniques  presented  in  this 
study  are  similar  in  that  they  are  all  time-series  methods.  Can  demand 
projections  be  based  on  non-time  series  models  as  well,  for  example 
relating  production  rates  and  back-orders?  If  so,  what  extrinsic  factors 
are  readily  available  for  use? 
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►►►  Can  less  restrictive  forecasting  methods  that  are  not 
bounded  by  conventional  weighting  schemes  be  used  more  effectively?  One 
such  method  has  been  proposed  by  DESC  which  incorporates  pattern 
recognition  techniques  through  neural  networks.  Although  this  type  of 
application  would  require  additional  research,  it  is  our  opinion  that  this 
method  could  be  included  as  a  forecasting  technique  within  a  multiple 
forecasting  model. 


►►►  There  is  seme  indication  of  possible  problems  associated 
with  the  MAD  under  a  multiple  forecasting  system.  Do  better  methods  exist 
for  estimating  this  essential  data  element  used  in  SAMMS  requirements 
computat ions? 


►►  The  effectiveness  of  the  prototype,  with  respect  to 
forecasting  accuaracy  and  system  impact,  should  be  evaluated  by  the  RFWG 
prior  to  full  implementation. 


37 


APPENDIX  A 

This  Appendix  supports  discussion  on 
recurring  and  nonrecurring  demand  base 
analysis.  Section  IX. C. 2,  page  21. 
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APPENDIX  B 

This  Appendix  supports  discussion  on  PERMES 
safety  level  analysis  for  the  system  inpact. 
Section  IX. E,  page  31. 
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Appendix  C 


PROJECTED  DATA  REQUIREMENTS 
FOR  MULTIPLE  FORECASTING  SYSTEM 


Projected  system  input  requirements  are  the  following: 


FIELDNAME  BYTES 

Defense  Supply  Center  (DSC)  1 

National  Stock  Number  (NSN)  13 

Item  Category  Code  (ICC)  1 

Supply  Status  Code  (SSC)  1 

Age  of  Item  Code  (AIC)  1 

Forecast  Basis  Code  (FBC)  1 

Administrative  Lead  Time  (ALT)  3 

Production  Lead  Time  (PLT)  3 

Procurement  Cycle  Months  (PCM)  3 

RECURRING  DEMAND:  Past  16  Quarters  9  each 

NONRECURRING  DEMAND:  Past  16  Quarters  9  each 


In  addition  the  system  would  likely  maintain  the  following  histories 

Selected  Forecast  Technique  Code:  Past  16  Quarters  1  each 
Actual  Forecast  History:  Past  16  Quarters  9  each 

Additional  inputs  and  histories  may  be  required. 
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